Classical Iterative Methods

posed on a finite dimensional Hilbert space V ∼= R equipped with an inner product (·, ·). Here A : V 7→ V is an symmetric and positive definite (SPD) operator, f ∈ V is given, and we are looking for u ∈ V such that (1) holds. The direct method to solve (1) is to form A−1 or the action of A−1. For example, the Gaussian elimination or LU factorization still remains the most commonly used methods in practice. It is a black-box as it can be applied to any problem in principle. However it requires the huge computational cost which, in the worst case, is O(N). For a sparse matrix, it may require less operations but still too expensive for large scale problems. For general dense matrices, a matrix-vector product requires O(N) operations and the solver would at least scale like O(N). When A is sparse, the nonzero entries of A is O(N) and the basic matrix-vector product reduces to O(N) operation. Then it is desireable to design optimally scaled solvers, say, with O(N) or O(N logN) computational cost. Namely computing A−1f is just a few number of Ax. To this end, we first introduce a basic residual-correction iterative method and study classic iterative methods. On the other hand, we should mention that the state-of-the-art of direct solvers can achieve the nearly linear complexity for certain structured sparse matrices; see for example [1].